The history is unclear.
Textual evidence represented by certain manuscripts and fragments from the Dead Sea caves makes it reasonably certain that the last of the canonical Hebrew writings had been completed several decades before the time Alexander the Great (356-323 B.C.) began his conquests in the Near East. The process by which they were accepted as canon was more protracted, however. Only when they had been circulated, read, and assessed favorably by comparison with the spirituality of the Torah were they accorded general canonicity. Hence, the distinction between canonical and apocryphal writings came as much through usage and general consent on the part of orthodox as in any other manner.
Earlier scholars suggested the so-called "Council of Jamnia" held in Palestine in 100 A.D. was responsible for drawing up a list of Old Testament books suitable for use by the faithful. However, subsequent studies have thrown considerable doubt upon the historicity of such a council, at the same time showing that the Jewish authorities of that period considered their noncanonical writings to be more of an obstacle than a help to devotion.
(The Origin of the Bible: Newly Updated by F. F. Bruce, J. I. Packer, Philip W. Comfort, and Carl F. H. Henry, 2020. Old Testament and New Testament Apocrypha by R. K. Harrison, Page 85.)
Where did the order of the books in the Hebrew Bible originate?
Hypothesis: The order reflects the order in which books were added to the canon
Some say the grouping reflects the gradual development of OT canon, so the cohesion is a historical accident. This hypothesis places the closing of the canon of The Prophets about the 3rd century B.C. The canon of the Hagiographa was not closed until the Jewish synod of Jamnia or Jabneh about A.D. 90. A broader canon had been accepted by the Greek-speaking Jews of Alexandria and was embodied in the Septuagint. This, plus the Essene fondness for the pseudonymous apocalypses are responsible for the fluidity of the OT canon in patristic Christianity.
Hypothesis: The order is based on groupings by literary character
Others say the books were grouped by literary character. The Law has histories covering the period from Creation to Moses, the Prophets has histories from Joshua to the end of the monarchy, and the Hagiographa has the half-narrative Daniel and histories of the Babylonian exile and return. Chronicles is last among the histories, a summary of the whole biblical narrative from Adam to the return.
(The Origin of the Bible: Newly Updated by F. F. Bruce, J. I. Packer, Philip W. Comfort, and Carl F. H. Henry, 2020. Biblical Languages by Larry Walker. Section One: The Authority and Inspiration of the Bible. Page 5-6; The Canon of the Old Testament by R. T. Beckwith. Page 56-61)
Development over Time
The distinctions of "The Prophets" and "Hagiographa" had to be solidified over a long period of time as a succession of prophets added to the Scripture, the prophetic gift was withdrawn for enough time for the books to be considered complete, and the books were collected and organized.
It is clear the canon of the Prophets was not closed when Chronicles was written, for it quotes not Samuel and Kings but their sources: the fuller prophetic histories. The early sections of Joshua and Samuel are very old, but so are the early sections of Psalms, Proverbs, and Chronicles. The latest elements in Daniel, Esther, and Ezra-Nehemiah are competing for the end of Old Testament history with Ezekiel, Haggai, Zechariah, and Malachi. The Hagiographa is younger overall, but the overlap is considerable. When the prophetic gift was withdrawn for some time, their number may have been seen as complete and the collection was organized.
Daniel 9:2 refers to a collection of sacred writings forming. The prologue to the Greek translation of Ecclesiasticus towards the end of the 2nd century refers to the Law, the Prophets, and "the others that have followed in their steps," or in other words, a section less old that is still being organized. Christ refers to the third as the Psalms in Luke 24:44, as does Philo in De Vita Contemplativa. Jesus' saying of "from the blood of Abel to the blood of Zechariah" probably implies Chronicles as the last book of the Old Testament, "from Genesis 4:3-15 to 2 Chronicles 24:19-22." The Prophets and Hagiographa were probably organized by 165 B.C. Also, the count of the books was known by the time of Josephus and the Greek translation of Jubilees, so that implies that the order is equally old, so maybe around the 1st century B.C.
2 Maccabees 2 describes a great crisis in the history of the canon - books were lost on account of the Maccabean Revolt or the Maccabaean war of liberation from the Syrian persecutor Antiochus Epiphanes, but they were recovered. Antiochus was hostile toward Scripture in 1 Maccabees 1:56.
Judas knew the prophetic gift had ceased (1 Maccabees 9:27), so he likely gathered the scattered Scriptures and sorted them with an order and number, giving us the first firm division of the Prophets and Hagiographa but also the traditional order and number of books within.
Though Josephus says the books have been accepted as canonical from time immemorial, individual attestation can be provided for the canonicity of nearly all books from writings of 1st century A.D. or earlier, even 4 out of 5 disputed books (not Song of Songs, maybe because of its brevity).
If the canon of the Old Testament was known by Jesus' day, why is the canon said to be closed by the synod of Jamnia? There's a few reason. The rabbinical literature disputes about 5 of the books, some of which were settled then.
(The Origin of the Bible: Newly Updated by F. F. Bruce, J. I. Packer, Philip W. Comfort, and Carl F. H. Henry, 2020. Biblical Languages by Larry Walker. Section One: The Authority and Inspiration of the Bible. Page 5-6; The Canon of the Old Testament by R. T. Beckwith. Page 56-61)
Also the Qumran discoveries show the apocalyptic pseudepigrapha to have been cherished and perhaps reckoned canonical by the Essenes. They were more likely viewed as an Essene appendix to the standard Jewish canon than an integral part of it. There are allusions to this appendix in Philo's account of the Therapeutae (De Vita Contemplativa 25) and in 2 Edras 14:44-48. Also discovered at Qumran, the Essenes had reckoned some of the Hagiographa as canonical and did so before the 2nd century rivalry began.
Since the Septuagint includes Apocrypha, the Alexandrian canon of Jews living in Alexandria is theorized to be broader than the Palestinian canon of Jews living in Palestine. The Alexandrian canon matches the Palestinian canon, shown by Philo of Alexandria's writings. The divisions are the same, and inspiration is only attributed to the canonical books, not the Apocrypha.
In the Septuagint manuscripts, the Prophets and Hagiographa have been rearranged by Christian hands in a non-Jewish manner and intermingled with the Apocrypha.
(The Origin of the Bible: Newly Updated by F. F. Bruce, J. I. Packer, Philip W. Comfort, and Carl F. H. Henry, 2020. Biblical Languages by Larry Walker. Section One: The Authority and Inspiration of the Bible. Page 5-6; The Canon of the Old Testament by R. T. Beckwith. Page 62)
The count of the books
The books are 24, but they could be made 22 when Ruth was appended to Judges and Lamentations to Jeremiah to match the length of the Hebrew alphabet. The number 24 is first recorded in Esdras 14:44-48, about A.D. 100. The number 22 is first recorded in Josephus's Contra Apion 1.8 or the Greek translation of the book of Jubilees (1st century B.C.).
(The Origin of the Bible: Newly Updated by F. F. Bruce, J. I. Packer, Philip W. Comfort, and Carl F. H. Henry, 2020. The Canon of the Old Testament by R. T. Beckwith. Pages 61.)
The Order of Books
The Law (comprises the Pentateuch)
Ends with the genealogy of David
The Hagiographa, The Writings, Sacred Writings